Building a Reinforcement Learning Trading Model

Clean financial tick data in preparation for training

Student must load data, view data, fill missing values, and plot filled data

Training Feature / State representation selection

Students must filter the HOLC data down to just Close (filtering). Student must use this close data to calculate 20-day bollinger bands (indicators)

Normalize data for input into a neural network

Students must normalize each training feature using SkLearn Standard Scaler (centers data with unit variance)

Split dataset into training and test sets

Students must split the dataset in half, use first half for training, and the second half for test. They must display the datasets to ensure they do not overlap. Students must convert dataframes to numpy arrays (formatting for keras usage)

Write a custom DQN architecture in keras

Students have to build their own DQN from scratch using keras - layer sizes, model type, optimizer type, and learning rate is provided for them.

Define the action policy for a DQN agent

Students must write their own action-selection policy utilizing an ε-greedy exploration-exploitation technique, and a q-value based action selector.

Define the experience replay (i.e. DQN fitting) process

Students must write their own experience replay function, where they define a mini-batch of memory, update target q-values using the Bellman equation, and fit the DQN model to the targeted values.

Define an RL Agent

All above methods must be components of the custom “Agent” class

Get the current state of the environment

Students must write their own get_state helper function which returns a 2-day sate representation (i.e. today’s Close and Bollinger Bands + yesterday's Close and BBs)

Implement & run the training loop

Students must fill in the training skeleton with real code. This includes getting the current and next state, performing inverse transforms with the normalizers to get true price data, defining rewards for different actions (buy and sell), updating the agent memory, and running the experience replay for every mini-batch of memory.

Implement & run the testing loop

Students must fill out the test loop skeleton and figure out how to put the Agent in test mode (i.e. how to load the trained model from a saved keras checkpoint).